Repository navigation
perf(server): index a new worktree in ~40s, not 6–9 min, by starting from the main checkout's index - #178
Closed
raimonade wants to merge 1 commit into
Closed
perf(server): index a new worktree in ~40s, not 6–9 min, by starting from the main checkout's index#178raimonade wants to merge 1 commit into
raimonade wants to merge 1 commit into
Conversation
|
Author
|
apologies, meant to send clanker to do the pr on my fork |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every agent that starts in a new worktree triggers a full code index of that worktree from nothing. On a large repo (16.5k files, ~10k indexed, ~210k symbols) that took 443–542 s per worktree in my server traces, and ~870 MB of index each. The agent usually finished its work before the index did, while two of those builds competed with everything else for CPU and memory, and the index was deleted again once the thread settled.
The worktree's sources are almost identical to its main checkout's, which usually already has a published index. Index paths are checkout-relative and every file record carries a content hash, so that index is valid for the worktree except for the files that differ.
What changed
buildIndexInWorker(apps/server/src/project/knowledge/codeIntelligenceWorker.ts): when a checkout has no published generation and is a linked git worktree, the build copies the main checkout's published database into its first generation, the same way a successor already copies its predecessor (FICLONE, so a reflink on APFS/btrfs), then runs the usual incrementalsync. Only files whose hash differs are re-parsed.git rev-parse --git-dir --git-common-dir; a worktree of a bare repository has none and builds from scratch as before..gitignore/codegraph.json), and a ranked listing when it is over the cap. Otherwise the build starts from scratch, as today.Evidence
Regression test:
starts a linked worktree's first index from its main checkout'sinCodeIntelligence.test.tsuses a realgit worktree add. The worktree lists only its new file, so a symbol from an unlisted-but-checked-out file can only be in its index if it came from the main checkout's.vp test run CodeIntelligence.test.ts CodeIndexReactor.test.ts codeIndexLocks.test.ts:Test Files 3 passed (3), Tests 103 passed | 5 skipped (108).Real repository:
buildIndexInWorkercalled directly from a script on an APFS clone of a live main-checkout index in a temp directory (M-series Mac, load average ~20 from other work during the run, one run each):SDK phase timings (
CODEGRAPH_SYNTH_TIMINGS=1) for the seeded build:sync-scoped: 581ms (10011 paths, 9999 tracked),sync-failed-ref-retry: 4969ms (11666 refs)against97964ms (71362 refs)for the from-scratch build.vp run check:changed: format, lint, knip, graphql and typecheck pass. Its test step was skipped locally because the project-memory addon needscargo, which this machine lacks; the test files above were run directly instead.Details
.gitignoreedit, before the main checkout's next build) is not used as a seed; that worktree builds from scratch, exactly as before.git rev-parseoutput and existing path helpers; it is untested there.ELI5
When an agent opens a copy of a project to work in, T3 used to re-read the whole project from scratch to build its code map, which took longer than the agent's work. Now it starts from the map it already has for the original project and only re-reads the few files that are different.